NAVAL  HEALTH  RESEARCH  CENTER 


AN  EVALUATION  OF  THE  CLINICAL 
EFFECTIVENESS  OF  TELEMEDICINE 
MEDICAL  PROVIDERS’ PERSPECTIVE 


G.  E.  Larson 
R.  G.  Burr 
D.  M.  Pearsall 
J.  Silva 


Report  No.  98-13 


,  DTIC  quality  INSPECTED  1 


Approved  far  public  release:  distribution  unlimited. 


NAVAL  HEALTH  RESEARCH  CENTER 
P.O.BOX  85122 

SAN  DIEGO,  CALIFORNIA  92186  -  5122 

NAVAL  MEDICAL  RESEARCH  AND  DEVELOPMENT  COMMAND 
BETHESDA,  MARYLAND 


19980 831  033 


An  Evaluation  of  the  Clinical  Effectiveness  of 
Telemedicine:  Medical  Providers’  Perspective 


Gerald  E.  Larson 
Ralph  G.  Burr 
Dianna  M.  Pearsall 
John  Silva 

Naval  Health  Research  Center 
Medical  Information  Systems  and 
Operations  Research  Department 
P.  O.  Box  85122 
San  Diego,  CA  92186-5122 


Report  No.  98-13  was  supported  by  the  Office  of  Naval  Research  under  Work  Unit 
63706N.M0096-6812.  The  views  expressed  in  this  article  are  those  of  the  authors  and  do  not 
reflect  the  official  policy  of  position  of  the  Department  of  the  Navy,  Department  of  Defense,  or 
the  U.S.  Government.  Approved  for  public  release;  distribution  is  unlimited. 


ABSTRACT 


Despite  the  growing  use  of  telemedicine  technology  in  civilian  and  military  health  care, 
relatively  little  work  has  been  performed  in  the  area  of  evaluation.  To  address  this 
deficiency,  a  preliminary  set  of  telemedicine  evaluation  instruments  (primarily  surveys) 
was  developed  and  pilot  tested.  Surveys  for  health  care  providers  were  administered 
aboard  three  Navy  carrier  groups  and  one  remote  shore  station.  A  separate  questionnaire, 
designed  for  the  medical  specialist  consultants  that  were  contacted  via  telecommun¬ 
ications,  was  administered  at  the  consultant  sites.  Results  suggest  that  providers  view 
telemedicine  as  a  highly  effective  tool,  and  that  telemedicine  consultations  have  their 
greatest  impact  on  treatment  (as  opposed  to  diagnosis).  The  usefulness  of  “basic” 
telemedicine  technologies,  such  as  telephone  and  e-mail,  was  strongly  supported. 
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INTRODUCTION 


Rapid  advances  in  information  technology  have  allowed  health  care  delivery 
services  to  be  expanded  through  telemedicine.1  Generally  defined,  telemedicine  is  the 
use  of  communications  technologies  to  support  medical  care  at  a  remote  location.  Some 
specific  examples  of  current  telemedicine  use  include  a  telephone  call  to  91 1  requesting 
instructions  for  administering  artificial  respiration,  a  paramedic  transmitting  an 
electrocardiogram  from  an  ambulance  while  en  route  to  a  hospital  emergency  room,  and 
an  interactive  video  transmission  via  satellite  allowing  a  physician  specialist  to  advise 
during  a  surgical  procedure  being  performed  in  another  country.  These  scenarios 
illustrate  the  purpose  of  telemedicine:  to  provide  immediate  and  expert  health  care  of  the 
highest  quality  to  an  individual  in  a  remote  location. 

Telemedicine  has  been  in  use  and  its  development  progressing  for  nearly  40 
years.  As  early  as  1959,  telemedicine  was  used  to  establish  an  interactive  television  link 
for  psychiatric  consultation  between  the  Nebraska  Psychiatric  Institute  in  Omaha  and  the 
Norfolk  State  Mental  hospital  over  100  miles  away.2  In  1968  an  interactive  television 
microwave  link  (which  included  an  electrocardiograph,  stethoscope,  and  voice 
transmission  capability)  was  established  between  Massachusetts  General  Hospital  in 
Boston  and  a  medical  station  at  the  city’s  Logan  Airport.3  During  the  early  1970s  only  15 
telemedicine  sites  received  federal  funding,4  by  the  1990s  telemedicine  had  begun  a 
period  of  rapid  growth.  Precipitated  by  significant  clinical  need,  the  medical  community 
currently  recognizes  telemedicine  as  more  than  a  sophisticated  means  of  communication 
and  exploits  this  technology  as  a  new  diagnostic  and  therapeutic  modality.  Medical 
specialties  as  diverse  as  psychiatry,  radiology,  dermatology,  and  cardiology  are  using 
telecommunications  to  eliminate  geographic  boundaries  between  a  physician  specialist 
and  patient.5-9 

At  present,  the  U.S.  military  is  a  widely  acknowledged  leader  in  many  aspects  of 
telemedicine.10  Telemedicine  has  particular  relevance  for  the  U.  S.  Navy;  ships  deployed 
to  remote  and  isolated  sites  of  conflict  often  have  few  resources  for  managing  a  wide 
range  of  medical  conditions.  Compounding  these  resource  limitations  is  the  fact  that  the 
majority  of  the  medical  departments  on  Navy  ships  are  headed  by  an  independent  duty 
corpsman  (IDC).  EDCs,  though  not  medical  doctors,  are  responsible  for  providing  all 
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aspects  of  primary  health  care  services  on  the  ship.  Given  the  challenges  that  an  IDC 
provider  must  routinely  face,  telecommunication  can  be  an  important  source  of  support 
for  difficult  diagnosis  and  treatment  decisions.  Not  only  does  the  patient  benefit,  but  the 
Navy  avoids  costly  medical  evacuations  (medevacs)  to  larger  ship  or  shore  facilities  (e.g., 
research  shows  that  ships  with  EDC  providers  have  more  medevacs  than  ships  with 
physician  providers.11) 

There  is  evidence  that  telemedicine  in  the  Navy  has  the  potential  to  keep  patient 
evacuations  at  an  absolute  minimum.  In  1983,  Nice11  conducted  a  9-month  study  of  all 
U.S.  Navy  surface  ships  (N  =  354),  Pacific  Fleet  submarines  (N  =  42),  and  all  ships  of  the 
Military  Sealift  Command  (N  =  54)  to  determine  the  need  for  telemedicine  capabilities 
aboard  ship.  This  study  showed  that  in  28%  of  all  medevacs,  the  senior  medical 
department  representative  indicated  that  there  was  a  significant  probability  that  the 
medevacs  could  have  been  prevented  if  they  had  had  the  capability  to  transmit  medical 
data  via  telecommunications.11  This  estimate  (28%)  was  subsequently  replicated  in  1997 
following  an  independent  study  by  the  Center  for  Naval  Analysis  (CNA).12 

In  recent  years,  the  Navy  has  implemented  telemedicine  aboard  several  aircraft 
carriers,  as  well  as  shore  facilities,  such  as  the  National  Naval  Medical  Center  in 
Bethesda,  MD  and  Naval  Medical  Center,  San  Diego,  CA.  The  Army  has  also  recognized 
the  potential  for  expanding  and  improving  the  quality  of  health  care  through  telemedicine 
technologies  and  has  played  a  major  role  in  the  military  telemedicine  revolution.13  In 
early  1 996  the  Army  expanded  telemedicine  support  to  include  the  Bosnia  theater  of 
operations.  Among  the  primary  modes  of  available  military  telecommunication  support 
are  teleradiology  (x-ray  images,  digitized  by  scanning,  and  transmitted  to  a  radiologist  for 
consultation),  video  teleconferencing  (full  motion  video  of  a  patient  sent  to  a 
consultant/specialist),  and  store  and  forward  images  (captured  in  digital  format  and  sent 
via  the  Internet  for  diagnosis  and  treatment  advice). 

The  need  for  evaluation.  In  1994  and  1995,  the  Commanding  Officer  of  the 
Naval  Medical  Information  Management  Center  issued  the  Navy  Telemedicine 
Initiative14  and  the  Telemedicine  Prototype  Project,15  citing  the  need  for  standardization 
and  evaluation  of  naval  telemedicine  technology.  In  addition,  a  number  of  recent 
research  articles  have  noted  the  lack  of  research  evaluating  the  effectiveness  of 
telemedicine.16"18  In  response  to  these  concerns,  the  Naval  Health  Research  Center 
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(NHRC)  in  San  Diego,  CA,  initiated  a  telemedicine  evaluation  project.  The  initial  effort 
of  this  project  was  to  develop  a  questionnaire  to  assess  medical  care  providers’ 
experiences  using  existing  telemedicine  systems  aboard  three  aircraft  carriers:  the 
George  Washington ,  the  Enterprise ,  the  Theodore  Roosevelt,  and  at  the  remote  shore  site 
at  McMurdo  Station,  Antarctica.  A  separate  questionnaire,  designed  for  the  medical 
specialist  consultants  that  were  contacted  via  telecommunications,  was  also  tested.  The 
objectives  of  the  present  report  were  to  (1)  pilot  test  and  evaluate  the  medical  provider 
and  consultant  questionnaires;  (2)  summarize  providers’  responses  regarding  the  use  and 
effectiveness  of  a  telemedicine  consultation  for  patient  diagnosis,  treatment,  and 
disposition;  and  (3)  match  provider  and  consultant  questionnaires  in  an  effort  to  compare 
and  contrast  their  respective  evaluations  of  specific  telemedicine  consults.  This  study 
was  the  first  in  a  planned  series  to  determine  whether  telecommunications  between  health 
care  providers  and  medical  consultants/specialists  were  effective  in  supporting  diagnosis 
and  treatment  during  medical  visits. 

DESIGN  AND  METHODOLOGY 

Two  separate  telemedicine  questionnaires  were  developed  at  the  NHRC  in 
consultation  with  representatives  of  the  National  Naval  Medical  Center  and  John  Hopkins 
University,  Baltimore,  MD.  The  questionnaires  were  designed  to  assess  provider  experiences 
and  consultant  experiences,  respectively,  with  a  special  emphasis  placed  on  the  role  of 
telemedicine  in  diagnosis  and  treatment.  Questionnaire  formats  were  heavily  influenced  by 
Dillman's19  total  design  methodology  for  mail  questionnaires. 

1.  Provider  Assessment  Questionnaire.  The  provider  questionnaire  solicits  health  care 
providers’  comments  on  each  telemedicine  transmission  Of  the  25  questions,  18  are  closed- 
ended  and  7  are  open-ended.  The  provider  questionnaire  was  designed  to  document  the 
reason(s)  for  the  medical  transmission;  whether  a  diagnosis,  treatment  plan,  or  disposition  had 
been  made  prior  to  the  transmission;  and  whether  the  telemedicine  contact  led  to  a  change  in 
diagnosis,  treatment  plan,  or  disposition  Five  additional  items  are  included  to  identify  the 
mode  of  transmission,  the  type  of  information  transferred,  and  ratings  of  the  timeliness  and 
ease  of  transmission  Last,  the  providers  are  asked  to  rate  the  telemedicine  transmission’s 
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effectiveness  in  providing  necessary  information,  and  to  rate  whether  the  transmission  assisted 
their  delivery  of  health  care. 

2-  Consultant  Assessment  Questionnaire.  The  consultant  questionnaire  (21  open- 
ended  questions  and  7  closed-ended)  was  designed  to  obtain  information  on  each  telemedicine 
transmission  at  the  consulting  locations.  The  majority  of  consultant  questions  mirror  those 
asked  of  the  Provider,  to  allow  comparisons.  The  primary  difference  between  these  two 
questionnaires  is  that  the  medical  consultants  are  also  asked  about  their  satisfaction  with  the 
quality  of  information  received  from  the  telemedicine  transmission 

Data  Collection.  Provider  surveys  were  distributed  aboard  the  three  aircraft 
carriers,  the  George  Washington,  the  Enterprise,  the  Theodore  Roosevelt,  and  at  the 
remote  shore  site  at  McMurdo  Station,  Antarctica.  During  the  same  time  period, 
consultant  surveys  were  distributed  at  the  National  Naval  Medical  Center  (the  consultant 
site  for  the  carriers)  and  at  Naval  Medical  Center,  San  Diego  (the  consultant  site  for 
McMurdo  Station).  Through  this  data  collection  strategy,  matching  provider  and 
consultant  evaluations  for  each  patient  could  be  obtained,  allowing  a  comparison  between 
the  two  perspectives. 

Data  Analysis  Techniques.  Upon  receipt,  the  numerical  data  obtained  from  the 
closed-ended  items  on  the  questionnaires  were  coded  and  entered  into  a  database  for  statistical 
analysis.  Then,  simple  frequency  distributions  were  created  for  the  provider  and  consultant 
responses,  the  characteristics  of  the  populations,  and  available  telecommunication 
technologies.  Next,  the  data  were  examined  in  various  ways  to  determine  the  impact  of 
telemedicine  communications  on  the  initial  diagnosis,  treatment  plans,  and  patient  disposition 
associated  with  each  case. 


RESULTS  AND  DISCUSSION 

The  first  objective  of  this  study  was  to  pilot  test  and  evaluate  the  provider  and 
consultant  questionnaires.  Within  limitations,  both  of  these  questionnaires  were 
determined  to  be  effective  information-gathering  instruments  based  on  the  data  received 
(described  as  follows).  The  content  and  length  were  deemed  appropriate,  and  a  clear 
picture  of  most  cases  emerged. 
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The  second  objective  of  this  study,  to  summarize  medical  providers’  responses 
regarding  the  use  and  effectiveness  of  telemedicine  consultations,  was  examined  by 
considering  a  number  of  questionnaire  items,  beginning  with  items  addressing  which  of 
the  various  telecommunication  modalities  were  used  and  which  type  of  information  was 
transmitted.  Providers  returned  94  completed  surveys.*  Table  1  shows  the  various 
telecommunication  modalities  along  with  the  frequency  and  percentage  of  usage  by  the 
medical  care  providers.  Provider  responses  show  that  televideo  and  the  Internet  were 
used  more  than  any  other  modality.  Values  for  televideo  usage,  however,  probably 
exceed  normal  levels  because  medical  providers  had  been  specifically  encouraged  to  use 
televideo  technology  shortly  before  the  present  study  began.  Telephone  communication 
was  also  frequently  used.  While  these  usage  results  are  constrained  by  equipment 
availability  on  the  ships  in  the  study,  the  findings  are  nevertheless  consistent  with  past 
research.  For  example,  studies  by  CNA  and  NHRC  have  consistently  demonstrated  the 
advantages  of  “low-end”  telemedicine  technologies,  such  as  e-mail  and  telephones. 12i 6,20 

Table  1 .  Telecommunication  Modalities,  Frequency,  and  Percentage  Used 


N 

% 

Televideo 

44 

35 

Internet 

44 

35 

Telephone 

27 

21 

Composite  Health  Care  System 

7 

6 

Radiophone 

3 

2 

T  eleradiographic 

2 

1 

Total 

127 

100 

Table  2  shows  the  type  of  data  or  information  that  were  transmitted  by  medical 
care  providers  to  medical  specialist  consultants.  The  most  frequent  type  of  data  or 
information  transmissions  involved  still  images,  followed  by  conversation,  and  live 
images  (tele video).  These  results  are,  of  course,  fairly  predictable  from  the 
telecommunication  modality  usage  rates  in  Table  1. 


’Response  rates  vary  by  survey  question,  particularly  since  some  items  allowed  respondents  to  select 
multiple  options. 
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Table  2.  Type  of  Telecommunication  Data  or  Information  Transmitted 


Still  images 

Conversation 

Live  images 

Records/test  results 

Radiologic  images 

Other 

Total 


N 

% 

39 

27 

31 

22 

25 

17 

24 

17 

23 

16 

2 

1 

144 

100 

Ninety-five  percent  of  the  requested  telemedicine  consultations  asked  for  the 
assistance  of  a  specialist.  Table  3  shows  the  medical  specialties  from  which  assistance 
was  requested.  The  medical  specialty  with  the  most  requests  for  assistance  was 
dermatology  (29%),  followed  by  orthopedics  (11%),  radiology  (9%),  and  ear/nose/throat 
(8%) .  These  results  support  those  of  an  Army  study  involving  a  peacekeeping  unit  in 
Macedonia  which  found  that  dermatology  and  orthopedics  were  the  specialties  most  often 
consulted  by  Army  practitioners.13  Since  these  specialties  also  experience  high 
consultation  rates  across  other  studies,1116,20  there  is  noteworthy  consistency  as  to  which 
medical  specialties  are  contacted  most  often  via  telemedicine. 
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Table  3.  Medical  Specialties  Consulted  via  Telecommunication 


Dermatology 

N 

25 

% 

29 

Orthopedics 

9 

11 

Radiology 

8 

9 

Ear/Nose/Throat 

7 

8 

Urology 

6 

7 

Neurosurgery 

5 

6 

Ophthalmology 

4 

5 

Cardiology 

3 

4 

Psychiatry 

3 

4 

Gynecology 

2 

2 

Oral  Surgery 

2 

2 

Gastroenterology 

1 

1 

Hand  Surgery 

1 

1 

Infectious  Disease 

1 

1 

Internal  Medicine 

1 

1 

Rheumatology 

1 

1 

Other 

6 

7 

Total 

85 

100 

Clinical  Impact.  Medical  care  providers  indicated  that  the  primary  reason  for  initiating  a 
telemedicine  consultation  was  to  develop  or  confirm  a  treatment  plan.  Specifically,  there  were 
substantially  more  requests  for  treatment  assistance  (46%)  than  for  help  with  diagnosis  (32%)  or 
decisions  on  a  patient’s  disposition  (16%).  Consistent  with  these  findings,  39%  of  treatment 
plans  changed  as  a  result  of  telemedicine  consultations,  while  diagnosis  changed  in  only  9%  of 
the  cases.  Our  finding  of  a  substantial  telemedicine  impact  on  treatment  is  supported  by  Army 
research  in  Macedonia,  where  66%  of  treatment  protocols  were  changed  as  a  result  of 
telemedicine  consultations.13  These  specific  Navy  and  Army  findings,  when  combined  with 
prior  research,1116  strongly  suggest  that  the  major  impact  of  telemedicine  is  on  patient  treatment 
rather  than  diagnosis  or  disposition. 

In  addition,  medical  care  providers  were  asked,  “overall,  how  effective  or  ineffective  was 
this  telemedicine  consultation  in  assisting  you  with  delivering  health  care  to  your  patient?” 
Eighty-four  percent  said  that  the  consultation  was  very  effective,  and  the  remaining  providers 
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(16%)  said  that  the  consultation  was  somewhat  effective.  None  of  the  medical  providers  felt 
that  the  consultations  were  ineffective. 

Provider/Consultant  Contrasts.  The  third  objective  of  the  study  was  to  match  provider 
and  consultant  questionnaires  for  each  case  in  an  effort  to  further  evaluate  all  aspects  of 
telemedicine  consultations.  Unfortunately,  the  number  of  provider/consultant  questionnaire 
matches  was  reduced  by  several  difficulties,  including  a  low  questionnaire  return  rate  from 
consultants  (N=34);  incomplete  consultant  questionnaires,  specifically,  missing  data  in  the  date 
field;  and  inconsistency  in  terminology  and/or  description  of  medical  encounters  between 
providers  and  consultants.  Because  of  these  difficulties  only  a  small  percentage  of  the  medical 
encounters  had  complete  and  matched  provider  and  consultant  questionnaires.  For  this  reason, 
provider  responses  were  the  primary  source  of  telemedicine  data  for  the  current  effort. 
Nevertheless,  several  impressions  can  be  gleaned  from  the  limited  consultant  data.  These  are 
presented  not  as  firm  results,  but  rather  as  trends  worthy  of  exploration  in  subsequent  work. 

For  example,  although  too  few  consultant  questionnaires  were  received  to  support 
detailed  analyses,  it  was  apparent  even  from  our  small  sample  that  many  consultants  had 
mistaken  impressions  about  providers’ diagnosis  and  treatment  progress.  Several  consultants 
wrote  that  no  prior  diagnosis  existed  at  the  time  of  the  telemedicine  consultation,  whereas 
providers  indicated  that  a  diagnosis  had  already  been  made.  Interestingly,  mistaken  consultant 
views  about  the  status  of  a  case  were  also  found  in  the  previously  cited  Army  study  of 
telemedicine  operations  in  Macedonia.13  In  at  least  4  separate  cases,  Army  hospital  consultants 
felt  that  an  evacuation  was  avoided  by  a  consult,  whereas  the  medical  provider  was  actually  not 
contemplating  evacuation  at  all.  Taken  together,  these  results  suggest  that  consultants  may  be  a 
relatively  unreliable  source  of  information  about  the  clinical  impact  of  telemedicine.  This,  of 
course,  by  no  means  diminishes  the  value  of  the  medical  advice  that  consultants  offer.  It  simply 
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reflects  the  fact  that  consultants  have  only  partial  knowledge  of  each  case  and  may  thus 
occasionally  make  incorrect  assumptions  about  preconsult  case  status. 

Technical  Issues.  All  telemedicine  technologies  were  viewed  as  relatively  simple  to  use, 
with  the  occasional  exception  of  video  teleconferencing  (VTC).  In  one  case,  10  separate 
attempts  were  required  to  establish  a  VTC  link-up  for  a  consultation,  wasting  valuable  time  and 
creating  frustration.  No  similar  problems  were  found  with,  for  example,  store  and  forward 
transmissions.  Although  VTC  problems,  including  poor  image  quality,  were  the  major  source 
of  technical  difficulties  noted  in  the  current  survey  responses,  these  problems  may  ease  as 
technology  improves. 


RECOMMENDATIONS 


Several  recommendations  are  suggested  in  light  of  the  present  study’s  telemedicine 
survey  results.  First,  while  the  current  sample  was  sufficient  to  draw  a  limited  number  of  broad 
conclusions,  larger  samples  will  be  needed  for  finer  grained  data  analysis.  Large  samples  are 
particularly  important  if  one  is  to  examine  the  effectiveness  of  telemedicine  as  a  function  of 
individual  medical  specialties  or  individual  patient  conditions.  In  the  present  study,  only 
dermatology  specialists  received  more  than  10  consults,  and  the  median  number  of  consults  for 
all  specialties  was  approximately  one.  These  numbers  are  too  small  for  examining  many 
critical  issues,  such  as  the  usefulness  of  diverse  telemedicine  devices  within  and  across 
specialties. 

An  alternative  to  large  samples  would  involve  small,  targeted  studies  focusing  on 
particular  specialties,  conditions,  or  technologies.  Indeed,  the  latter  approach  may  be  the  most 
viable  because  it  addresses  more  highly  focused  research  issues  (e.g.,  “does  VTC  help 
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psychologists  treat  shipboard  personnel  with  stress  symptomatology?”).  Information  from 
small  but  in-depth  studies  could  then  be  aggregated  with  broad-band  surveys  to  provide  an 
overall  perspective  with  both  breadth  and  detail. 

Second,  even  if  larger  samples  are  sought  it  can  be  difficult  to  obtain  an  adequate 
response  rate,  particularly  from  consultants.  It  is  understandable  that  the  consultants,  who  are 
practicing  physicians,  may  be  less  compliant  regarding  additional  paperwork  than  the 
providers,  who  are  often  relatively  junior  enlisted  personnel.  Nevertheless,  it  would  be 
desirable  to  obtain  a  higher  consultant  response  rate  and  possible  strategies  to  reach  this  goal 
should  be  explored.  Perhaps  consultants  should  be  assessed  through  one-time  interviews  in 
which  a  retrospective,  overall  summary  of  telemedicine  experiences  is  obtained.  Case-by-case 
information  could  then  still  be  obtained  from  medical  care  providers,  who  are  the  real  experts 
on  what  transpired  with  each  patient. 

Finally,  while  the  current  effort  has  focused  on  changes  in  diagnosis  and  treatment,  a 
truly  complete  evaluation  of  telemedicine  must  also  acknowledge  its  benefits  for  training. 
Continuing  medical  education  has  always  been  a  proposed  use  of  telemedicine,  and 
telemedicine-based  training  should  continue  to  be  explored  as  an  adjunct  to  other  state-of-the- 
art  tools,  such  as  computer-based  patient  treatment  simulations  and  expert-system21medical 
decision  aids.  Various  educational  or  training  uses  for  telemedicine  hold  great  promise  for 
military  health  care  providers. 
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